A Multimodal Infant Behavior Annotation for Developmental Analysis of Demonstrative Expressions

We have obtained the valuable findings about the developmental processes of demonstrative expression skills, which is concerned with the fundamental commonsense of human knowledge, such as to get an object and to catch someone’s attention. We have already developed a framework to record genuine spontaneous speech of infants. We are constructing a multimodal infant behavior corpus, which enables us to elucidate human commonsense knowledge and its acquisition mechanism. Based on the observation of the corpus, we proposed a multimodal behavior description for observation of demonstrative expressions. We proved that the proposed model has the nearly 90% coverage in an open test of the behavior description task. The analysis using the model produced many valuable findings from multimodal viewpoints; for example, the change of ‘line of sight’ from ‘object to person’ to ‘person to object’ means that the infant has obtained a better way to catch someone’s attention. Our intention-based analysis provided us with an infant behavior model that may apply to a likely behavior simulation system.